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(57) ABSTRACT 

System and method for optimization of a design associated 
with a response function, using a hybrid neural net and 
support vector machine (NN/SVM) analysis to minimize or 
maximize an objective function, optionally subject to one or 
more constraints. As a first example, the NN/SVM analysis 
is applied iteratively to design of an aerodynamic compo- 
nent, such as an airfoil shape, where the objective function 
measures deviation from a target pressure distribution on the 
perimeter of the aerodynamic component. As a second 
example, the NN/SVM analysis is applied to data classifi- 
cation of a sequence of data points in a multidimensional 
space. The NN/SVM analysis is also applied to data regres- 
sion. 

8 Claims, 14 Drawing Sheets 
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HYBRID NEURAL NETWORK AND 
SUPPORT VECTOR MACHINE METHOD 
FOR OPTIMIZATION 

ORIGIN OF THE INVENTION 5 

This application is a Divisional of U.S. Ser. No. 10/043, 
044 now U.S. Pat. No. 6,961,719, filed 7 Jan. 2002, issued 
1 Nov. 2005. The invention disclosed herein was made by an 
employee of the U.S. Government and may be manufactured l o 
and used by or for the Government for governmental pur- 
poses without payment of any royalties for such manufac- 
ture and use. 

FIELD OF THE INVENTION 15 

This invention relates to design optimization, using a 
hybrid neural network and support vector machine approach 
to construct a response surface that models a selected 
objective function. 20 

BACKGROUND OF THE INVENTION 

Considerable advances have been made in the past two 
decades in developing advanced techniques for numerical 2 5 
simulation of fluid flows in aerodynamic configurations. 
These techniques are now mature enough to be used rou- 
tinely, in conjunction with experimental results, in aerody- 
namic design. However, aerodynamic design optimization 
procedures that make efficient use of these advanced tech- 30 
niques are still being developed. 

The design of aircraft components, such as a wing, a 
fuselage or an engine, involves obtaining an optimal com- 
ponent shape that can deliver the desired level of component 
performance, subject to one or more constraints (such as 35 
maximum weight or cost) that the component(s) must sat- 
isfy. Aerodynamic design can be formulated as an optimi- 
zation problem that requires minimization of an objective 
function, subject to constraints. Many formal optimization 
methods have been developed and applied to aerodynamic 40 
design. These include inverse design methods, adjoint meth- 
ods, sensitivity derivative-based methods and traditional 
response surface methodology (RSM). 

Inverse design methods in aerodynamics are used to 
provide a component that responds in a pre-selected manner, 45 
for example, an aircraft wing that has a prescribed pressure 
distribution. The known inverse methods do not account for 
certain fluid parameters, such as viscosity, and are used in 
preliminary design only. 

Adjoint methods provide a designer with the gradient of 50 
the objective function. One advantage of this method is that 
the gradient information is obtained very quickly. However, 
where several technical disciplines are applied simulta- 
neously, it is often difficult to perform design optimization 
using this method; each discipline requires a different for- 55 
mulation. It is also difficult and expensive to quickly evalu- 
ate the effects of engineering tradeoffs, where the applicable 
constraints may be changed several times. It is also not 
possible to use existing experimental data or partial or 
unstructured data in the design process. 60 

A sensitivity derivative-based method typically requires 
that a multiplicity of solutions, with one parameter varied at 
a time, be obtained to compute a gradient of the objective 
function. The number of computations required grows lin- 
early with the number of design parameters considered for 65 
optimization, and this method quickly becomes computa- 
tionally expensive. This method is also sensitive to noise 
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present in the design data sets. As with an adjoint method, 
it is not possible to use existing experimental data or partial 
or unstructured data in the design process. 

RSM provides a framework for obtaining an optimal 
design, using statistical procedures, such as regression 
analysis and design of experiments. Traditional RSM uses 
low-degree regression polynomials in the relevant design 
variables to model the variation of an objective function. 
The polynomial model is then analyzed to obtain an optimal 
design. Several polynomial models may have to be con- 
structed to provide an adequate view of the design space. 
Addition of higher degree polynomials will increase the 
computational cost and will build in higher sensitivity to 
noise in the data used. 

Artificial neural networks (“neural nets” herein) have 
been widely used in fields such as aerodynamic engineering, 
for modeling and analysis of flow control, estimation of 
aerodynamic coefficients, grid generation and data interpo- 
lation. Neural nets have been used in RSM-based design 
optimization, to replace or complement a polynomial -based 
regression analysis. Current applications of neural nets are 
limited to simple designs involving only a few design 
parameters. The number of data sets required for adequate 
modeling may increase geometrically or exponentially with 
the number of design parameters examined. A neural net 
analysis requires that the design space be populated with 
sufficiently dense simulation and/or experimental data. Use 
of sparse data may result in an inaccurate representation of 
the objective function in design space. On the other hand, 
inefficient use of design data in populating the design space 
can result in excessive simulation costs. Capacity control is 
critical to obtain good generalization capability. In some 
preceding work, this problem was alleviated by using a 
neural net to represent the functional behavior with respect 
to only those variables that result in complex, as opposed to 
simple, variations of the objective function; the functional 
behavior of the remaining variables was modeled using low 
degree polynomials. This requires a priori knowledge to 
partition the design variables into two sets. 

FIG. 1 graphically illustrates results of applying a simple 
NN analysis to a one-parameter model, namely, an approxi- 
mation to the second degree polynomial y=2-(0.5-x ) 2 at 
each of 3 pairs of training values (curve A) and at each of 5 
pairs of training values (curve B). Use of more than the 
minimum number (3) of training pairs clearly improves the 
fit over the domain of the variable x. It is theoretically 
possible that only Q+l spaced apart training value pairs are 
needed to completely specify a Qth degree polynomial (for 
example, Q= 6 ). However, because of the presence of noise, 
the theoretical minimum number of training value pairs is 
seldom sufficient to provide an acceptable fit. 

Use of neural network (NN) analysis of a physical object, 
in order to optimize response of the object in a specified 
physical environment, is well known. An example is opti- 
mization of a turbine blade shape, in two or three dimen- 
sions, in order to reproduce an idealized pressure distribu- 
tion along the blade surface, as disclosed by Rai and 
Madavan in “Aerodynamic Design Using Neural Net- 
works”, AIAA Jour., vol. 38 (2000) pp. 173-182. NN analy- 
sis is suitable for multidimensional interpolation of data that 
lack structure and provides a natural structure in which a 
succession of numerical solutions of increasing complexity, 
or increasing fidelity to a real world environment, can be 
represented and optimized. NN analysis is especially useful 
when multiple design objectives need to be met. 

A feed-forward neural net is a nonlinear estimation tech- 
nique. One difficulty associated with use of a feed- forward 
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neural net arises from the need for nonlinear optimization to 
determine connection weights between input, intermediate 
and output variables. The training process can be very 
expensive when large amounts of data need to be modeled. 

In response to this, a support vector machine (SVM) 5 
approach, originally applied in statistical learning theory, 
has been developed and applied. Support vector machine 
analysis allows use of a feature space with a large dimen- 
sion, through use of a mapping from input space into feature 
space and use of a dual formulation of the governing to 
equations and constraints. One advantage of an SVM 
approach is that the objective function (which is to be 
minimized to obtain the coefficients that define the SVM 
model) is convex so that any local minimum is also a global 
minimum; this is not true for many neural net models. 15 
However, an underlying feature space (polynomial, Gauss- 
ian, etc.) must be specified in a conventional SVM approach, 
and data resampling is required to implement model hybrid- 
ization. Hybridization is more naturally, and less expen- 
sively, applied in a neural net analysis. 20 

What is needed is a machine learning algorithm that 
combines the desirable features of NN analysis and of SVM 
analysis and does not require intimate a priori familiarity 
with operational details of the object to be optimized. 
Preferably, the method should automatically provide a char- 25 
acterization of many or all of the aspects in feature space 
needed for the analysis. 


4 

FIG. 12 graphically illustrates data classification accord- 
ing to the invention. 

DESCRIPTION OF BEST MODES OF THE 
INVENTION 

Consider a feed-forward neural network 21 having an 
input layer with nodes 23 -m (m=l, . . . , 5), a hidden layer 
with nodes 25 -n (n=l, 2, 3), and an output node 26, as 
illustrated schematically in FIG. 2. The first input layer node 
23-1 has a bias input value 1, in appropriate units. The 
remaining nodes of the input layer are used to enter selected 
parameter values as input variables, expressed as a vector 
p=(pi, . . . , p^), with M^l. Each node 2 5-n of the hidden 
layer is associated with a nonlinear activation function 


of a weighted sum of the parameter values p m , where C nm is 
a connection weight, which can be positive, negative or zero, 
linking an input node 23-m with a hidden layer node 25 -n. 
The output of the network 21 is assumed for simplicity, 
initially, to be a single-valued scalar, 


SUMMARY OF THE INVENTION 

The invention meets these needs by providing a hybrid of 
NN analysis and SVM analysis, referred to as NN/SVM 
analysis herein. In one embodiment, NN/SVM analysis 
begins with a group of associated, independent input space 
coordinates (parameter values), maps these coordinates into 
a feature space of appropriately higher dimension that 
includes a computed set of combinations (e.g., powers) of 
the input space coordinates with the assistance of the input 
and hidden layers of an NN, constructs an inner product 
formalism for the coordinates in feature space, obtains a 
solution to a minimization problem to compute Lagrange 
multiplier values that define the SVM, and returns to input 
space to complete a solution of the problem. 

BRIEF DESCRIPTION OF THE DRAWINGS 


FIG. 2 illustrates a conventional three-layer NN, with an 
input layer, a hidden layer and an output layer that receives 
and combines the resulting signals produced by the hidden 
layer. 

It is known that NN approximations of the format set forth 
in Eqs. (1) and (2) are dense in the space of continuous 
functions when the activation functions <D„ are continuous 
sigmoidal functions (monotonically increasing functions, 
with a selected lower limit, such as 0, and a selected upper 
limit, such as 1). Three commonly used sigmoidal functions 
are 

<I>(z)=l/{l +exp (— z) (3 A) 


30 


35 


40 


45 


FIG. 1 graphically illustrates an improvement in match of 
a polynomial, where an increased number of training pairs 
is included in a simple NN analysis. 

FIG. 2 is a schematic view of a three-layer feed- forward 
neural net in the prior art. 

FIG. 3 is a schematic view of a two-layer feed-forward 
NN/SVM system according to the invention. 

FIG. 4 is a flow chart of an overall procedure for prac- 
ticing the invention using an NN/SVM system. 

FIGS. 5, 6 and 7 graphically illustrate generalization 
curves obtained for a fifth degree polynomial, a logarithm 
function and an exponential function, respectively, using a 
hybrid NN/SVM analysis and 11 training values. 

FIGS. 8A/8B/8C are a flow chart for an RSM procedure 
used in practicing the invention. 

FIGS. 9A1-9C2 graphically illustrate evolution of an 
airfoil and corresponding pressure distribution obtained 
from an iterative NN/SVM analysis. 65 

FIGS. 10 and 11A/11B illustrate data classification in two 
dimensions. 


<J>(z)=(l +tanh(z)}/2, 

(3B) 

<I>(z)={(jt+2 -tan ~(z) }/2 ji, 

(3C) 

M 

(4) 


Z — Gun ■ p m . 


m=l 


Other sigmoidal functions can also be used here. In the 
context of design optimization, a trained NN represents a 
response surface, and the NN output is the objective func- 
tion. In multiple objective optimization, different NNs can 
be used for different objective functions. A rapid training 
algorithm that determines the connection weights C„ m and 
coefficients D„ is also needed here. 

The approach set forth in the preceding does reasonably 
well in an interpolative mode, that is, in regions where data 
points (parameter value vectors) are reasonably plentiful. 


50 


55 


60 
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However, this approach rarely does well in an extrapolative 
mode. In this latter situation, a precipitous drop in estimation 
accuracy may occur as one moves beyond the convex hull 
defined by the data point locations. In part, this is because 
the sigmoidal functions are not the most appropriate basis 
functions for most data modeling situations. Where the 
underlying fiinction(s) is a polynomial in the parameter 
values, a more appropriate set of basis functions is a set of 
Legendre functions (if the parameter value domain is finite), 
or a set of Laguerre or Hermite functions (if the parameter 
value domain is infinite). Where the underlying function(s) 
is periodic in a parameter value, a Fourier series may be 
more appropriate to represent the variation of the function 
with that parameter. 

Two well known approaches are available for reducing 
the disparity between an underlying function and an activa- 
tion function. A first approach, relies on neural nets and uses 
appropriate functions of the primary variables as additional 
input signals for the input nodes. These functions simplify 
relationships between neural net input and output variables 
but require a priori knowledge of these relationships, includ- 
ing specification of all the important nonlinear terms in the 
variables. For example, a function of the (independent) 
parameter values x and y, such as 

h {x,y)=a ■x 2 +b -x-y+c-y^+d-x+e ■ y+f (5) 

where a, b, c, d, e and f are constant coefficients, would be 
better approximated if the terms x, y, x 2 , x y and y 2 are all 
supplied to the input nodes of the network 21. However, in 
a more general setting with many parameters, this leads to 
a very large number of input nodes and as-yet-undetermined 
connection weights C nm . 

A second approach, referred to as a support vector 
machine (SVM), provides a nonlinear transformation from 
the input space variables p m into a feature space that contains 
the original variables p m and the important nonlinear com- 
binations of such terms (e.g., (pj 2 , (pi)(p 2 ) 3 (p^) 2 and exp 
(p 2 )) as coordinates. For the example function h(p 1? p 2 ) set 
forth in Eq. (5), the five appropriate feature space coordi- 
nates would be p l5 p 2 , (p x ) 2 , Pi'p 2 and (p 2 ) 2 * Very high 
dimensional feature spaces can be handled efficiently using 
kernel functions for certain choices of feature space coor- 
dinates. The total mapping between the input space of 
individual variables (first power of each parameter p m ) and 
the output space is a hyperplane in feature space. For a 
model that requires only linear terms and polynomial terms 
of total degree 2 (as in Eq. (5)), in the input space variables, 
the model can be constructed efficiently using kernel func- 
tions that can be used to define inner products between 
vectors in feature space. However, use of an SVM requires 
a priori knowledge of the functional relationships between 
input and output variables. 

The mapping between the input space parameters and the 
output function is defined using a kernel function and certain 
Lagrange multipliers. The Lagrange multipliers are obtained 
by maximizing a function that is quadratic and convex in the 
multipliers, the advantage being that every local minimum is 
also a global minimum. By contrast, a neural net often 
exhibits numerous local minima of the training error(s) that 
may not be global minima. However, several of these local 
minima may provide acceptable training errors. The result- 
ing multiplicity of acceptable weight vectors can be used to 
provide superior network generalization, using a process 
known as network hybridization. A hybrid network can be 
constructed from the individual trained networks, without 
requiring data re-sampling or similar expensive techniques. 
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An attractive feature of a neural net, vis-a-vis an SVM, is 
that the coordinates used in a feature space do not have to be 
specified (e.g., via kernel functions). However, use of an 
SVM, in contrast to use of a neural net, allows one to 
5 introduce features spaces with a large number of dimen- 
sions, without a corresponding increase in the number of 
coefficients. 

A primary contribution of the present invention is to 
provide a mechanism, within the NN component, for deter- 
mining at least the coordinate (parameter) combinations 
needed to adequately define the feature space for an SVM, 
without requiring detailed knowledge of the relationships 
between input parameters and the output function. 

15 FIG. 3 is a schematic view of an NN/SVM system 31, 
including an NN component and an SVM component, 
according to the invention. The system 31 includes input 
layer nodes 33-/ (i=l, . . . , 5) and hidden layer nodes 35 -j 
(=1, 2, 3). FIG. 3 also indicates some of the connection 
20 weights associated with connections of the input layer 
terminals and the hidden layer terminals. More than one 
hidden layer can be provided. The hidden layer output 
signals are individually received at an SVM 37 for further 
processing, including computation of a training error. If the 
25 computed training error is too large, one or more of the 
connection weights is changed, and the (changed) connec- 
tion weights are returned to the NN component input ter- 
minals for repetition of the procedure. Optionally, the SVM 
37 receives one or more user- specified augmented inner 
30 product or kernel prescriptions (discussed in the following), 
including selected combinations of coordinates to be added, 
from an augmentation source 38. 

FIG. 4 is a flow chart illustrating an overall procedure 
according to the invention. In step 41, the system provides 
35 (initial) values for connection weights C nm for the input 
layer-hidden layer connections. These weights may be ran- 
domly chosen. The input signals may be a vector of param- 
eter values p=(p 1? . . . , $ M ) (M=5 in FIG. 3) in parameter 
space. In step 42, output signals from the hidden layer are 
40 computed to define the feature space for the SVM. The NN 
component of the system will provide appropriate combi- 
nations of the parameter space coordinates as new coordi- 
nates in a feature space for the SVM (e.g., u 1 =p 1 , u 2 =p 2 , 
„ c U 3 =P! 2 , U 4 =p 1 p 2 , u 5 =p 2 2 , from Eq. (5)) 

In step 43, feature space inner products that are required 
for the SVM are computed. In step 43A, user-specified 
feature space coordinates and corresponding inner products 
and kernel functions are provided. Note that the feature 
5Q space is a vector space with a corresponding inner product. 

In step 44, a Lagrange functional is defined and mini- 
mized, subject to constraints, to obtain Lagrange multiplier 
values for the SVM. See the Appendix for a discussion of a 
Lagrange functional and associated constraints. In step 45, 
55 the NN connection weights and the Lagrange multiplier 
coefficients are incorporated and used to compute a training 
error associated with this choice of values within the 
NN/SVM. 

In step 46, the system determines if the training error is no 
60 greater than a specified threshold level. If the answer to the 
query in step 46 is “no”, the system changes at least one 
connection weight, in step 47, preferably in a direction that 
is likely to reduce the training error, and repeats steps 42-46. 
If the answer to the query in step 46 is “yes”, the system 
65 interprets the present set of connection weights and 
Lagrange multiplier values as an optimal solution of the 
problem, in step 48. 



US 7,293,001 B1 


7 

Note that steps 42-48 can be embedded in an optimization 
loop, wherein the connection weights are changed according 
to the rules of the particular optimization method used. 

The hybrid NN/SVM system relies on the following 
broadly stated actions: (1) provide initial random (or other- 
wise specified) connection weights for the NN; (2) use the 
activation function(s) and the connection weights associated 
with each hidden layer unit to construct inner products for 
the SVM; (3) use the inner products to compute the 
Lagrange multiplier values; (4) compute a training error 
associated with the present values of the connection weights 
and Lagrange multiplier values; (5) if the training error is too 
large, change at least one connection weight and repeat steps 
(2) -(4); (6) if the training error is not too large, accept the 
resulting values of the connection weights and the Lagrange 
multiplier values as optimal. 

This method has several advantages over a conventional 
SVM approach. First, coordinates that must be specified a 
priori in the feature space for a conventional SVM are 
determined by the NN component in an NN/SVM system. 
The feature space coordinates are generated by the NN 
component to correspond to the data at hand. In other words, 
the feature space provided by the NN component evolves to 
match or correspond to the data. A feature space that evolves 
in this manner is referred to as “data -adaptive.” The feature 
space coordinates generated by the NN component can be 
easily augmented with additional user- specified feature 
space coordinates (parameter combinations) and kernel 
functions. 

Second, use of activation functions that are nonlinear 
functions of the connection weights in the NN component 
reintroduces the possibility of multiple local minima and 
provides a possibility of hybridization without requiring data 
resampling. 

The feature spaces generated by the NN hidden layer can 
be easily augmented with high-dimensional feature spaces 
without requiring a corresponding increase in the number of 
connection weights. For example, a polynomial kernel con- 
taining all monomials and binomials (degrees one and two) 
in the parameter space coordinates can be added to an inner 
product generated by the SVM component, without requir- 
ing any additional connection weights or Lagrange multi- 
plier coefficients. 

The NN/SVM system employs nonlinear optimization 
methods to obtain acceptable connection weights, but the 
weight vectors thus found are not necessarily unique. Many 
different weight vectors may provide acceptably low train- 
ing errors for a given set of training data. This multiplicity 
of acceptable weight vectors can be used to advantage. If 
validation data are available, one can select the connection 
weight vector and resulting NN/SVM system with the 
smallest validation error. In aerodynamics, this requires 
additional simulations that can be computationally expen- 
sive. 

If validation data are not available, multiple trained NNs 
or NN/SVM systems can be utilized by creating a hybrid 
NN/SVM. A weighted average of N output signals from 
trained NN/SVMs in a hybrid NN/SVM is formed as a new 
solution. Where the weights are equal, if errors for the N 
individual output solutions are uncorrelated and individually 
have zero mean, the least squares error of this new solution 
is approximately a factor of N less than the average of the 
least squares errors for the N individual solutions. When the 
errors for the N individual output solutions are partly 
correlated, the hybrid solution continues to produce a least 
squares error that is smaller than the average of the least 
squares errors for the N individual solutions, but the differ- 
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ence is not as large. The N trained NN/SVMs used to form 
a hybrid system need not have the same architecture or be 
trained using the same training set. 

FIG. 5 graphically illustrates results of applying an 
5 NN/SVM analysis according to the invention to a six- 
parameter model, namely, an approximation to the fifth 
degree polynomial y=x(l-x 2 )(4-x 2 ). Data are provided at 
each of 1 1 training locations (indicated by small circles on 
the curve) in the domain of the variable x. After a few 
to iterations of an NN/SVM analysis, the 11 training values, 
(x A: ,y A: )=(x A: , x A: (l -X£ 2 )(4-X£ 2 )), provide the solid curve as a 
generalization, using the NN/SVM analysis. The dashed 
curve (barely visible in FIG. 5) is a plot of the original fifth 
order polynomial. 

15 FIG. 6 graphically illustrates similar results of an appli- 
cation of the NN/SVM analysis to a logarithm function, 
y=ln(x+4), using 11 training values. The solid curve is the 
generalization provided by the NN/SVM analysis. 

FIG. 7 graphically illustrates similar results of an appli- 
20 cation of the NN/SVM analysis to an exponential function, 
y=6*exp(-0.5*x 2 ), using 11 training values. The solid curve 
is the generalization provided by the NN/SVM analysis, 
using the 11 training values. 

The generalization in each of FIGS. 5, 6 and 7 is vastly 
25 superior to corresponding generalizations provided by con- 
ventional approaches. In obtaining such a generalization, the 
same computer code can be used, with no change of param- 
eters or other variables required. 

FIGS. 8 A, 8B and 8C are a flow chart illustrating the 
30 application of a response surface methodology (RSM) used 
in this invention to obtain an optimal cross-sectional shape 
of an airfoil, as an example, where specified pressure values 
at selected locations on the airfoil perimeter are to be 
matched as closely as possible. In step 81, a set of param- 
35 eters, expressed here as a vector p=(p 15 . . . , p^), is provided 
that adequately describes the airfoil cross-sectional shape 
(referred to as a “shape” herein), where M (iG) is a selected 
positive integer. For example, the airfoil shape might be 
described by (1) first and second radii that approximate the 
40 shape of the airfoil at the leading edge and at the trailing 
edge, (2) four coefficients that describe a tension spline fit of 
the upper perimeter of the airfoil between the leading and 
trailing edge shapes, and (3) four coefficients that describe a 
tension spline fit of the lower perimeter of the airfoil 
45 between the leading and trailing edge shapes, a total of ten 
parameters. In a more general setting, the number M of 
parameters may range from 2 to 20 or more. 

In step 82, initial values of the parameters, p=p0, are 
provided from an initial approximation to the desired airfoil 
50 shape. 

In step 83, optimal data values Pfeopt) (e.g., airfoil 
pressure values or airfoil heat transfer values) are provided 
at selected locations r^=(x Ar ,y Ar ,z^)(k=l, . . . , K) on the airfoil 
perimeter. 

55 In step 84, an equilateral M-simplex, denoted MS(p0), is 
constructed, with a centroid or other selected central location 
at p=p0, in M-dimensional parameter space, with vertices 
lying on a unit radius sphere. Each of the M+l vertices of the 
M-simplex MS(p0) is connected to the centroid, p=p0, by a 
60 vector Ap(m) (m=l, . . . , M+l) in parameter space. More 
than the M+l vertices can be selected and used within the 
M-simplex. For example, midpoints of each of the M(M+ 
l)/2 simplex edges can be added to the M+l vertices. These 
additional locations will provide a more accurate NN/SVM 
65 model. 

In step 85, a computational fluid dynamics (CFD) or other 
calculation is performed for an extended parameter value 
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set, consisting of the parameter value vectors p=pO and each 
of the M+l M-simplex vertices, p=p ver ,=pO+Ap(m), to 
obtain a calculated pressure distribution Pfep^) at each of 
the selected perimeter locations, r=r k for each of these 
parameter value sets. One hybrid NN/SVM is assigned to 5 
perform the analysis for all vertices in the M-simplex 
MS(pO) at each location r k . That is, a total of K NN/SVM 
systems are used to model the overall pressure dependence 
on the parameters p m . The calculated pressure distribution 
Pfepve/*) and/or the airfoil can be replaced by any other to 
suitable physical model, in aerodynamics or in any other 
technical field or discipline. Used together, the trained 
NN/SVM systems will provide the pressure distribution 
P(r^;p) for general parameter value vectors p. 

In step 86, a first objective function, such as 15 


k (6A) 

OBJ{p\ pO; 1) = ^ w k {P(r k ; p) - P{r k ; opt)} 2 . 


is introduced, where {w^} is a selected set of non-negative 
weight coefficients. 

In step 87, the minimum value of the first objective 25 
function OBJ(p;pO;l) and a corresponding parameter vector 
p=p(min) are determined for parameter vectors p within a 
selected sphere having a selected diameter or dilatation 
factor d, defined by Ip-pOl^d, with l<d= 10. The process is 
performed using a nonlinear optimization method. Other 30 
measures of extrapolation can also be used here. 

In step 88 , the system calculates a second objective 
function, which may be the first objective function or 
(preferably) may be defined as 


K 

OBJ(p\ pOi 2) = «*{/>(/•*; p; CFD ) - T\n ; opt)} 2 , 

k=l 


(6B) 


40 


where P^pjCFD) is a pressure value computed using a 
CFD simulation, for p=p(min) and p=pO. The system then 
determines if OBJ(p(min);pO;2)<OBJ(pO;pO;2) for the inter- 
mediate minimum value parameter vector, p=p(min). One 45 
can use the first objective function OBJ(p;pO;l), defined in 
Eq. ( 6 A), rather than the objective function OBJ(p;pO;2) 
defined in Eq. ( 6 B), for this comparison, but the resulting 
inaccuracies may be large. 

If the answer to the query in step 88 is “no” for the choice 50 
of dilatation factor d, the dilatation factor d is reduced to a 
smaller value d' (l<d'<d), in step 89, and steps 88 and 89 are 
repeated until the approximation pressure values {Pfep)}* 
for the extrapolated parameter value set provide an 
improved approximation for the optimal values for the same 55 
airfoil perimeter locations, r=r k . 

If the answer to the query in step 88 is “yes”, the system 
moves to step 90, uses the (modified) objective function and 
uses the intermediate minimum-cost parameter value set, 
p=p(min), which may lie inside or outside the M-simplex 60 
MS(p0) in parameter space. Minimization of the objective 
function OBJ(p;pO) may include one or more constraints, 
which may be enforced using the well known method of 
penalty functions. The (modified) objective function defini- 
tion in Eq. ( 6 A) (or in Eq. ( 6 B)) can be replaced by any other 65 
positive definite definition of an objective function, for 
example, by 


K 

OBJ(p\ p0 ) = Yj w k\P(r k -, p) - P{r k \ opt)\ q , 

k = 1 


(6C) 


where q is a selected positive number. 

If the original parameter value set p has an insufficient 
number of parameters, this will become evident in the 
preceding calculations, and the (modified) objective func- 
tion OBJ(p(min);pO) or OBJ(p(min);pO)* will not tend 
toward acceptably small numbers. In this situation, at least 
one additional parameter would be added to the parameter 
value set p and the procedure would be repeated. In effect, 
an NN/SVM procedure used in an RSM analysis will require 
addition of (one or more) parameters until the convergence 
toward a minimum value that is acceptable for an optimized 
design. 

In step 91, the system determines if the (modified) objec- 
tive function OBJ(p(min);pO)* is no greater than a selected 
threshold number (e.g., 1 or 10 -4 , in appropriate units). If the 
answer to the query in step 91 is “no”, a new M-simplex 
MS(p’0) is formulated, in step 92, with p , 0=p(min) as the 
new center, and steps 85-90 are repeated at least once. Each 
time, a new parameter value set, p=p(min), is determined 
that approximately minimizes the objective function OBJ(p; 
p'O). 

If the answer to the query in step 91 is “yes”, the system 
interprets the resulting parameter set, p=p(min), and the 
design described by this parameter set as optimal, in step 93. 
The method set forth in steps 81-93 is referred to herein as 
a response surface method. 

FIGS. 9A1-9C2 illustrate a sequence of partly-optimized 
designs for an airfoil, obtained using the invention, and 
compare each such design shape and corresponding airfoil 
pressure distribution to an target airfoil design shape and 
corresponding target airfoil pressure distribution. The objec- 
tive function is defined as mean square error between 
resulting and target pressure distribution at a sequence of 
selected locations on the airfoil perimeter. One begins in 
FIG. 9A1 with a curvilinear shape of approximately uniform 
thickness, which provides a pressure distribution p along the 
airfoil perimeter as illustrated graphically in FIG. 9A2. 
FIGS. 9B1 and 9C1 illustrate the results of second and 
fourth iterative applications of an NN/SVM analysis accord- 
ing to the invention, and FIGS. 9B2 and 9C2 graphically 
illustrate the pressure distributions corresponding to FIGS. 
9B1 and 9C1, respectively. Each iteration brings the result- 
ing airfoil shape and pressure distribution closer to the target 
shape and target pressure distribution. After a fourth itera- 
tion of the NN/SVM analysis, the airfoil shape, shown in 
FIG. 9C1, produces a pressure distribution, shown in FIG. 
9C2, that nearly precisely matches the target airfoil pressure 
distribution. Computations for this iterative sequence 
required about 8 minutes on a 16-processor SGI Origin 
computer. 

In a second embodiment, NN/SVM analysis is applied to 
data classification in a multi-dimensional vector space. In 
data classification, a discrimination mechanism must be 
determined that divides the data points into (at least) a first 
set of data points that satisfy a selected criterion, and a 
second set of data points that either do not satisfy the (first) 
criterion or that satisfy an inconsistent second criterion. FIG. 
10 illustrates a collection of first set data points (“x”) and 
second set data points (“o”) in two (parameter) dimensions 
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that are easily separated by a linear function of the two 
parameter coordinates, namely 

fi(x,y)ra-x+by-c= 0, (7) 

where a, b and c are selected real values, with at least one 5 
of a and b being non-zero: All data points in the first data set 
and in the second data set lie on opposite sides of the line 
(hyperplane) f^Xjy^O. Here, the data point separation is 
straightforward. 

FIGS. 11A and 11 B illustrate a collection of first set data 10 
points (“x”) and second set data points (“o”) that cannot be 
separated using a linear function of the two coordinates. An 
appropriate separation function may be 

f 2 (x,y)=(a-x+b y-c) 2 ±(d-x+e-g) 2 =l , (8) 15 

where a*d+b*e=0 and a, b, c, d, e and g are selected real 
values, not all zero. The choice of the plus (+) sign in Eq. (8) 
produces an ellipse, and the choice of a minus (-) sign in Eq. 

(8) produces a hyperbola. In this instance, one set of ^ 
appropriate coordinates for hyperplane separation in feature 
space is 


u x =x. 

(9A) 

«2 =y. 

(9B) 

u 3 =(a -x+b y-c) 2 , 

(9C) 

u 4 =(d-x+e y-g) 2 , 

(9D) 


in which the separating hyperplane in feature space becomes 30 

u 3 ±u 4 -l=0. (10) 

The power of an SVM resides, in part, in its use of a qth 
order polynomial kernel (as an example) for vectors a and 
p, such as 35 

^a,p)=(a-p+l)*, (11) 

where q is a selected positive integer (e.g., q=2), rather than 
requiring an a priori definition of the polynomial terms to be 
used, as in Eqs. (9A)-(9D). 40 

An advantage of the present invention, using NN/SVM 
analysis, over a conventional SVM analysis is that the 
kernel, such as the one given in Eq. (11), and the associated 
feature space need not be specified a priori; the appropriate 
feature space is automatically generated by the NN compo- 45 
nent of the NN/SVM system during the training process. 

FIG. 12 illustrates an application of the NN/SVM system 
to data classification, with M=2. Two classes of data that are 
separable, indicated as crosses and squares, are provided for 
the system. The exact boundary between the two classes is 50 
defined by first and second intersecting ellipses in two 
dimensions, with the major axes being oriented at 45° and at 
135° relative to an x-axis in an (x,y) region p defined by 

p={(x,y)\0^x^2.5, 0%^2.5}. (12) 

55 

Four hundred data points were randomly generated in this 
region and were first classified according to the exact 
boundaries. The boundaries were then removed, and only 
the locations of the data points were provided to the 
NN/SVM system. The resulting decision boundary gener- 60 
ated by the NN/SVM system is shown as a solid line in FIG. 

12. More generally, if M-parameter data points are provided, 
with Mi^2, the data separation surface or hyperplane will 
have dimension at most M-l. 

The NN/SVM system provides a perfect classification of 65 
the original data, with zero mis-assignments, without requir- 
ing any specification of kernel functions or feature spaces. 


Where the solid boundary line and the dotted boundary lines 
differ, no data points were located in the intervening regions 
between these boundaries. Provision of additional data 
points in one or more of these intervening regions would 
provide a resulting (solid) NN/SVM boundary line that is 
closer to the exact (dotted) boundary line. 

If r is a ratio of the sum of the absolute value of the 
intervening regions corresponding to the boundary lines 
mismatch, and the area of the square (6.25 units 2 in FIG. 12), 
the ratio r is a very small number that will tend toward zero 
as the number of data points (assumed to be approximately 
uniformly distributed) increases without bound. Addition- 
ally, r (defined as a percentage) represents the number of 
misclassifications (also expressed as a percentage) that an 
NN/SVM-generated boundary will produce on a very large 
test set. 


Appendix 

Examples of an NN analysis and of an SVM analysis are 
presented here. The invention is not limited to a particular 
NN analysis or to a particular SVM analysis. 

Consider an object, represented by a group of coordinates 
x=(x\ x 2 , . . . , x^), for which some physical feature or 
response of the object is to be optimized. The object may be 
a aircraft wing or turbine blade for which an ideal pressure 
distribution at specified locations on the object is to be 
achieved as closely as possible. The object may be a 
chemically reacting system with desired percentages of final 
compounds, for which total thermal eneigy output is mini- 
mized. The object may be represented at spaced apart 
locations or at spaced apart times by a group of independent 
coordinates, and an objective or cost function is presented, 
representing the response to be optimized. One or more 
constraints, either physical or numerical, are also set down, 
if desired. 

In an NN analysis, one relevant problem is minimizing 
empirical risk over a sum of linear indicator or characteristic 
functions 


N 


f{x, w) = W/V}, 

(=1 


(A-l) 


where 0 is an indicator or characteristic function, x is a 
coordinate vector and w is a vector of selected weight 
coefficients. Consider a training set of (N+l)-tuples (x^yj, 
(x 2 ,y 2 ), . . . , (x K ,y K ), where each x,.=(x/, x/, . . . , x/) is an 
N-tuple representing a vector and y \ is a scalar having only 
the values -1 or +1. 

The indicator function 0(z) has only two values, 0 and 1, 
and is not generally differentiable with respect to a variable 
in its argument. The indicator function 0(z) in Eq. (A-l) is 
often replaced by a general sigmoid function S(z) that is 
differentiable with respect to z everywhere on the finite real 
line, is monotonically increasing with z, and satisfies 

Lim 2 . j0 _ oo 5(z)=O, (A-2a) 


Lim x0+oo 5(z)=l. 


(A-2b) 
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Examples of suitable sigmoid functions include the follow- 
ing: 


S(z)=l/{l+exp(-az)}, 

5 

S(z)={ 1 +tanh(|3-z+%)]/2 
,S(z)={ Jt+2 -tan - 1 (5-z+e}/2jt, 

where a, (3 and 8 are selected positive values. The indicator 10 
sum f(x,w) in Eq. (A-l) is replaced by a modified sigmoid 


G(x, w) = Sf jr }. 


(A- 3) 15 


where S is a selected linear or nonlinear function. 

In order to minimize the empirical risk, one must deter- 
mine the parameter values w z that minimize an empirical risk 
functional 


20 


VM = Yi iyj - F(XJ ’ w))2 1 K ' 

7= 1 


(A- 4) 


30 

which is differentiable in the vector components w. One 
may, for example, use a gradient search approach to mini- 
mize R em/7 (w). The search may converge to a local mini- 
mum, which may or may not be a global minimum for the 
empirical risk. 35 

Assume, first, that the training data {(x 7 ,y 7 )} can be 
separated by an optimal separating hyperplane, defined by 


(vv-x-)-g=0, 


(A-5) 


where g partly defines the hyperplane. A separating hyper- 40 
plane satisfies 


O ■*,•)_?= i ty&i), 


(A- 6a) 




(A-6b) 45 


An optimal separating hyperplane maximizes the functional 

$(w/)=(wtv)/2, (A-7) 

with respect to the vector values w and the value g, subject 50 
to the constraints in Eqs. (A-6a)-(A-6b). Unless indicated 
otherwise, all sums in the following are understood to be 
over the index j=l, . . . , K). 

A solution to this optimization problem is given by a 
saddle point of a Lagrange functional 


55 


L(w, g, a) = (w-w)/ 2 - ^ -w)-g)- ( yj - 1)}. 

7=1 


(A- 8) 


60 


At a saddle point, the solutions (w,g,a) satisfy the relations 

dL/dg= 0, (A-9) 65 

(A- 10) 
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with the associated constraint 

<X/=0, (A-ll) 

Equation (A-9) yields the constraint 


K 


Yj a j' y j = 0 - 

7=i 


(A- 12) 


Equation (A- 10) provides an expression for the parameter 
vector w of an optimal hyperplane as a linear combination 
of vectors in the training set 

w=HyyafXj, (A-l 3) 

An optimal solution (w,g,a) must satisfy a Kuhn-Tucker 
condition 

a^l ((*/ w) ' (y,- 1 )=° (1=1, ■ ■ ■ , K)- (A- 14) 

Only some of the training vectors, referred to herein as 
“support vectors,” have non-zero coefficients in the expan- 
sion of the optimal solution vector w. More precisely, the 
expansion in Eq. (A- 13) can be rewritten as 

w=J.yyayx r (A-15) 

support vectors 

Substituting the optimal vector w back into Eq. (A- 8) and 
taking into account the Kuhn-Tucker condition, the 
Lagrange functional to be minimized is re-expressed as 


K K 

U.a) ='Yj a j- (1 / 2 )^ a ‘ ' a J -yryj- ( x i - x j)- 
7=1 i,7=i 


(A- 16) 


This functional is to be maximized, subject to the constraints 
expressed in Eqs. (A- 13) and (A- 14). Substituting the 
expression for optimal parameter vector w into Eq. (A- 14), 
one obtains 

(wx)-g=2ay (xyx)-g=0. (A-17) 

The preceding development assumes that the training set 
data {(x^y,)} are separable by a hyperplane. If these data are 
not separable by a hyperplane, one introduces non-negative 
slack variables X/G = l, • • • , K) and a modified functional 

<t>(w)=(w-w)+C-2X/, (A-l 8) 


subject to the constraints 

yA(wXj)-g) = l-Xj, (A- 19) 

where the (positive) coefficient C corresponds to an inter- 
penetration of two or more groups of training set (N+l)- 
tuples into each other (thus, precluding separation by a 
hyperplane). Repeating the preceding analysis, where the 
functional <D(w) replaces the term(w-w), an optimal solution 
(w,g,a) is found as before by maximizing a quadratic form, 
subject to the modified constraints 

20^=0., (A-20a) 

0^a fiC. (A-20b) 

Use of (only) hyperplanes in an input space is insufficient for 
certain classes of data. See the examples in FIGS. 11 A and 
11B. 


dL/dw= 0, 
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In a support vector machine, input vectors are mapped 
into a high dimension feature space Z through a selected 
nonlinear mapping. In the space Z, an optimal separating 
hyperplane is constructed that maximizes a certain A-margin 
associated with hyperplane separation. 

First, consider a mapping that allows one to construct 
decision polynomials of degree 2 in the input space. One 
creates a (quadratic) feature space Z having dimension 
M=N(N+3)/2, with coordinates 

Uj=x! (j=l, . . . , N: N coordinates) (A-21a) 

U j+N =x f (/'= 1, . . . , N; N coordinates) (A-21b) 


U j+2N =X l ' x 2> x i' x 3 > ■■■ , x N -i'X n , (N(N- 1)/2 coordi- 
nates). 


(A-21c) 


A separating hyperplane constructed in the space Z is 
assumed to be a second degree polynomial in the input space 
coordinates x^=l, . . . , N). 

By analogy, in order to construct a polynomial of degree 
k in the input coordinates, one must construct a space Z 
having of the order of N* coordinates, where one constructs 
an optimal separating hyperplane. For example, for k=4, the 
maximum number of coordinates needed in the space Z is 

max(k=4)=(N+k)U{N!k!} k= 4 (A-22) 

which is about 10 8 coordinates for a modest size input space 
of N=100 independent coordinates. 

For a quadratic feature space Z, one first determines a 
kernel function K of inner products according to 

(Ml i 'Ujjz ') = K(x J i >Xj2 ') = K(x J 2,Xj i ) (L 1 , £2=1, . . . ,N(N+ 

l)/2). (A-23) 

One constructs nonlinear decision functions 

/(x)=sgn{2a / -J^(x,x / )+^0 } (A-24) 

support vectors 

that are equivalent to the decision function <D(x) in Eq. 
(A-l 8). By analogy with the preceding, the coefficients aj are 
estimated by solving the equation 

W( a)=2a,-(l/2) Za-a ;-x i -x J -K(x i ,x j ), (A-25) 

with the following constraint (or sequence of constraints) 
imposed: 

(A-26a) 

a,^0. (A-26b) 

Mercer (1909) has proved that a one-to-one correspondence 
exists between the set of symmetric, positive definite func- 
tions K(x,y) defined on the real line that satisfy 

J J K(x,y]/{x]/(y)dx dy^O (A-27) 

for any L2-integrable function f(x) satisfying 

jf(x) 2 dx< co (A-28) 


and the set of inner products defined on that function space 
{f}. Thus, any kernel function K(x ;1 ,x ;2 ) satisfying condi- 
tions of the Mercer theorem can be used to construct an inner 
product of the type set forth in Eq. (A-23). Using different 
expressions for the kernel K(x ;1 ,x ;2 ), one can construct 
different learning machines with corresponding nonlinear 
decision functions. 

For example, the kernel function 


2ST(;c';c") = { (x '-x ")+ 1 } q 
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can be used to specify polynomials of degree up to q 
(preferably an integer). 

Much of the preceding development is taken from V. N. 
Vapnik, “An Overview of Statistical Learning Theory”, 
5 IEEE Trans. Neural Networks, vol. 10 (1999), pp. 988-999. 
The present invention provides a hybrid approach in which 
the input layer and hidden layer(s) of an NN component are 
used to create a data-adaptive feature space for an SVM 
component. As indicated in the preceding, the combined 
to NN/SVM analysis of the invention is not limited to the 
particular NN analysis or to the particular SVM analysis set 
forth in this Appendix. 
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What is claimed is: 


1 . A method for aerodynamic design optimization of an 
aerodynamic component, the method comprising: 
providing a computer that is programmed: 

20 (1) to provide a group of M parameters that define a 

design shape of a selected aerodynamic component, 
and to provide a vector p=p0 of initial values for each 
parameter in the group, where M is a selected integer 
iU; 

25 (2) to provide data point values and at least one 

numerical criterion for an optimal design at one or 
more selected location values of the aerodynamic 
component; 

(3) to provide an M-simplex in parameter space, cen- 

30 tered at a vector location p=p0 and having a selected 

diameter dO; 

(4) to provide a design function f k at each location value 
k on the aerodynamic component, depending on the 
choice of the parameter vector p, for the parameter 
vector pO and for each parameter vector correspond- 
ing to a vertex of the M-simplex, and for an 
expanded M-simplex centered at p0; 

(5) to provide a selected first objective function OBJ 

40 (p;p0; 1), dependent upon the parameter vector p and 

upon a difference between the optimal design data 
point value and a first design function data point 
value at one or more of the location values; 

(6) to determine a parameter vector p=p(min) within 

45 the expanded M-simplex for which the first obj ective 

function attains a minimum value; 

(7) to compute a selected second objective function 
OBJ(p;pO;2) for p=p(min) and for p=p0; 

(8) when OBJ(p(min);pO;2) is not less than OBJ(pO; 

50 p0;2), to provide a modified expanded M-simplex, 

with a modified diameter d’ satisfying d0<d’<d and to 
repeat steps (6) and (7) at least once; and 

(9) when OBJ(p(min);pO;2) is less than OBJ(pO;pO;2), 
to determine if OBJ(p(min);pO;2) is no greater than 

55 a selected threshold value; 


(10) when OBJ(p(min);pO;2) is greater than the thresh- 
old value, to provide a substitute M-simplex, cen- 
tered at p=p , 0=p(min) with the selected diameter dO, 
and an expanded substitute M-simplex, centered at 
p'O with the diameter d, and repeat steps (4), (5), (6), 
(7), (8) and (9) at least once; and 

(11) when OBJ(p(min);pO;2) is not greater than the 
threshold value, to interpret the parameter set p=p 
(min) as an optimal design set; and 

(12) constructing an aerodynamic component with a 
shape according to the optimal design set. 


(A-29) 
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2. The method of claim 1, further comprising choosing 
said first objective function to be 

K 

OBJ(p\ pO) = ^ Wk\fk(r k ; p) - fk{r k \ opt )\ q , 

jt=i 

where r k is one of said selected location values, 4(r^;p) is one 
of said design function data point values, f^r^opt) is one of 
said optimal design data point values, w* is a selected 
non-negative weight coefficient, q is a selected positive 
number, and K is a selected positive integer. 

3. The method of claim 1, further comprising choosing 
said first objective function to be the same as said second 
objective function. 

4. The method of claim 1, wherein said second objective 
function depends upon said second design function, upon 
said parameter vector p and upon a difference between said 
optimal design data point value and a data point value 
computed using computer simulation of a response of said 
design, at one or more of said location values. 
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5. The method of claim 1, further comprising choosing 
said design function to correspond to pressure on an airfoil 
at said selected locations on a perimeter of the airfoil. 

6. The method of claim 1, wherein said computer is 
5 further programmed: 

(13) to provide data point values and at least a second 
numerical criterion for a second optimal design at one 
or more selected location values; and 

(14) to cause said computer to apply steps (1) and (3)-(ll) 
to of claim 1 to obtain an optimal set of design parameters 

for the second optimal design, where the second opti- 
mal design has selected third and fourth objective 
functions that are independent of said first and second 
objective functions. 

15 7. The method of claim 1, further comprising selecting 

said aerodynamic component to be a turbine blade or 
compressor blade and said design to be a shape, viewed in 
plan view, of the blade. 

8. The method of claim 1, further comprising selecting 
20 said aerodynamic component to be an aircraft wing and said 
design to be a shape, viewed in plan view, of the wing. 



